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SUMMARY OF RESEARCH: 

Pilot Non-Conformance to Alerting System Commands 


This research project examined the effects of consonance between cockpit displays and alerting 
system as a technique to encourage pilots to conform to alerting system commands. An experiment 
used the task of collision avoidance during closely spaced parallel approaches as a case study, 
building upon previous experiments which identified instances of non-conformance and conflicts 
between the alerting criteria preferred by pilots, compared to that used by alerting systems. 

Using a workstation based, part-task simulator, each of 45 subjects completed 45 experiment 
runs. In each run, the subjects were told they were flying an approach. Their primary task was to 
keep their wings level despite turbulence through the use of a sidestick. The sidestick commands 
did not affect the path of the aircraft, however, so that consistent approach paths were be followed. 
Their secondary task was to indicate when an aircraft on a parallel approach is blundering towards 
them, as evidenced by the traffic display. Subjects were asked to press different buttons indicating 
whether they feel an avoidance maneuver is required by the traffic situation or not. At the 
completion of each run, subjects were asked to rate their confidence in their decision and, if 
appropriate, to rate the timeliness of automatic alerts when had been given. 

Three different automatic alert conditions were tested. The “No Automatic Alerts Given” 
condition is self-explanatory. In the “Automatic Alerts Based on NTZ Criteria” condition, an 
automatic alert was given when the NTZ criteria was triggered; this criteria is consistent with 
subject reactions in other studies, in which subjects were found to react, on average, when the 
other aircraft was 1350’ to the side of the own aircraft. In the “Automatic Alerts Based on MIT 
Criteria” condition, an automatic alert was given when the MIT criteria was triggered; this criteria 
was developed by Carpenter and Kuchar for parallel approaches to have better performance, at the 
cost of increased complexity and higher sensitivity to pilot non-conformance. 

Three displays were tested. The “Baseline Display” is based on a moving map display, with a 
top-down view, track-up orientation, iconic presentation of the other aircraft’s positions and a text 
presentation of the other aircraft’s altitude. The “NTZ Alert Criteria Shown” display added to the 
Baseline Display an explicit presentation on the display of the boundary delineating the NTZ 
criteria, allowing for quick comparison of the other aircraft’s position relative to this alert criteria. 
Likewise, the “MIT Alert Criteria Shown” display added to the Baseline Display an explicit 
presentation on the display of the boundary delineating the MIT criteria, allowing for quick 
comparison of the other aircraft’s position relative to this alert criteria 

The test matrix for this experiment was three dimensional, varying displays, alerts and traffic 
conflict scenarios. The test matrix is shown in Figure 1. It was unpaired by displays to reduce 
learning effects 
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Figure 1. Initial Experiment Test Matrix 
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Pilot Non-Conformance to Alerting System Commands 

Based on the conclusions of the results of the other, previous studies, several hypotheses were 
made entering into the experiment. First, it was hypothesized that subject reactions would be 
influenced by the display of an alerting criteria to have reactions more consistent with that alerting 
criteria. 

Results from the experiment generally support this hypothesis. Specific measurements taken 
during the experiment allowed a comparison of the timing of the subjects’ reactions to when the 
two different alert criteria would have triggered. In the conditions where the MIT criteria was 
explicitly shown on the traffic display, the subjects were found to have reactions closer in time to 
the time when the MIT alert criteria triggered. It should be noted, however, that the dominant 
effect in causing the subjects to react closer to the alerting criteria may be difficult to distinguish. 
Because subjects tended to react after the two alerting criteria typically triggered, it is difficult to 
discern whether the display of the MIT criteria encouraged subjects to react closer to the alerting 
criteria, or instead prompted them less specifically to react earlier without discrimination of specific 
effects. Regardless of causality, this display tended to result in earlier, safer decisions by the 
subjects. 

The presentation of the NTZ criteria on the traffic display, on the other hand was found to have 
fewer significant effects. This result was not unexpected, however, as the NTZ criteria was 
chosen to be an approximation to how subjects reacted in previous experiments using the Baseline 
display; thus, the explicit presentation of the NTZ criteria on the traffic display may have not 
emphasized a different reaction from the Baseline display. 

The second hypothesis was that subject reactions would be influenced by an automatic alert to 
have reactions more consistent with the underlying alert criteria, although to a lesser degree than 
for the displays. The first part of this hypothesis was generally supported by the results. Overall, 
subjects tended to react, on average, about 5 seconds after an automatic alert. According to the 
briefing, these reactions were requested to be based on their “best judgment”, and therefore do not 
represent solely their base reaction time. 

The different in time when the MIT criteria was triggered and when the subject reacted was 
significantly lower when an automatic alert was given based on the MIT criteria. This indicates 
that the presentation of an MIT automatic alert encouraged subjects to react sooner, on average, 
than they would have otherwise. However, an automatic alert based on the NTZ criteria was not 
found to encourage subjects to react closer to the NTZ automatic criteria. 

Where significant differences were found between alert conditions, they were not always 
smaller in magnitude than the differences found between display conditions. Therefore, while the 
alert condition effects did not have the same prevalence as the display condition effects, they were 
not always smaller as suggested in the second part of the hypothesis. 

An underlying assumption of this experiment is that subjects inherently would have reactions 
closest to the NTZ criteria unless encouraged to follow the higher performance MIT criteria 
through displays and alerts. The experiment results found this to also be true, with subjects’ 
reactions significantly closer to the NTZ criteria in most scenarios than to the MIT criteria. 

The final hypotheses examined the interaction between the explicit presentation of an alerting 
criteria on the traffic display and automatic alerts. It was hypothesized that “consonance” between 
the display of an alert criteria and the automatic alert — a condition where the displayed criteria 
matched the triggering criteria of the automatic alert, providing visual justification of the timing of 
the automation alert -- would shift the subject’s alerting strategy to be more consistent with the 
alerts and would increase subject confidence in the alerting system. Like, it was hypothesized that 
“dissonance” between the display of alert criteria and automatic alerts would shift the pilot’s 
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alerting strategy to be less consistent with the alerts and would have a negative effect on subjects’ 
confidence in the alerting system. 

Consonance was created in those blocks in the test matrix in which an NTZ-based automatic 
alert was given and the NTZ alert criteria was shown explicitly on the screen, and in which an 
MIT-based automatic alert was given and the MIT alert criteria was shown explicitly on the screen. 
Dissonance was created in those blocks in the test matrix in which an NTZ-based automatic alert 
was given and the MIT alert criteria was shown explicitly on the screen, and in which an MIT- 
based automatic alert was given and the NTZ alert criteria was shown explicitly on the screen. 

Analysis of Variance (ANOVA) on those conditions testing consonance and dissonance found 
several significant effects. Overall, consonance between the traffic display showing the MIT alert 
criteria and an MIT criteria -based automatic alert was found to have subject reactions closest to the 
MIT criteria. Likewise, dissonance between the traffic display showing the NTZ alert criteria and 
an MIT criteria -based automatic alert was found to have subject reactions the farthest to the MIT 
criteria, implying that the presentation of familiar information which can contradict high 
performance but non-intuitive automatic alerts should be prevented. 

Consonance between the traffic display showing the NTZ alert criteria and an NTZ criteria - 
based automatic alert created much closer agreement between the automatic alerts and the time of 
the subjects’ reactions than the identical consonant case involving the MIT alert criteria. This may 
indicate the continuing strong preference of the subjects for NTZ-type alert criteria, and highlights 
the difficulty in shaping subjects’ best judgment to agree with the underlying basis for the high- 
performance alerting system commands. 

In summary, these results support two conclusions. First, for detecting and reacting to a 
potential collision during parallel approaches, the disagreement between subject strategies and the 
underlying logic intended for automatic alerting systems can be substantial, highlighting the need 
for further investigation of pilot non-conformance to alerting system commands and its related 
effects - low pilot trust in the alerting system and increased pilot workload. 

Second, the effect of dissonant information on the cockpit displays, suggesting a different 
reaction to a hazard, may become increasingly substantial as alerting systems become more 
complex, thereby matching the pilots’ strategies less and requiring more explanatory information. 

A full description of this research will be documented in the Master’s thesis of Bal4zs Vandor, 
expected in the Summer of 1998. Complementing this research, an undergraduate special project 
is conducting a smaller experiment using the same simulator setup, examining the minimum base 
reaction time of subjects when order to follow automatic alerts exactly. The results of this special 
project are expected to provide a baseline to compare against the ‘best judgment’ reactions of the 
subjects in the research study documented here. 
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